Advertisement campaign simulator

ABSTRACT

A shadow ad can be evaluated by receiving an ad request, identifying at least one shadow ad and at least one actual ad based on the received ad request, generating an ad ranking by analyzing one or more criteria associated with the identified at least one shadow ad and the at least one actual ad, and selecting one or more of the identified at least one shadow ad and the at least one actual ad based on the ad ranking. Further, the at least one shadow ad can be associated with one or more campaign parameters corresponding to a shadow ad campaign. Additionally, the ad request can be received from an online advertising system.

CROSS REFERENCE TO RELATED APPLICATION

This application is a continuation application of, and claims priorityto, pending U.S. patent application Ser. No. 11/864,633, filed on Sep.28, 2007, and entitled “Advertisement Campaign Simulator”, which claimspriority to U.S. Provisional Patent Application No. 60/827,680, filed onSep. 29, 2006, and entitled “Advertisement Campaign Simulator”, theentirety of which is herein incorporated by reference.

BACKGROUND

The present disclosure relates to advertising in an online environmentand to simulating an online advertising campaign. The present disclosurefurther relates to conducting an advertising auction based on a shadowadvertising campaign and one or more actual advertising campaigns, andto strategies for evaluating the performance of a shadow advertisingcampaign based on one or more actual advertising campaigns.

Internet advertising has increased significantly with the expandingcommercialization of the World-Wide-Web. Advertising on the Internetbegan, at least in part, with advertisements (or “ads”) that werefeatured as content on advertisers' home pages. As with conventionalmedia, an Internet ad campaign can comprise one or more ads that aredisplayed through one or more outlets, such as web sites or individualweb pages.

An ad can serve one or more intended purposes, including commercial,charitable, educational, and entertainment purposes. Additionally, anonline ad can exist in a variety of forms, such as text, video, audio,image, or any combination thereof. Each display of an ad to a user isreferred to as an impression (or an “exposure”). Typically, animpression is generated when a web page including the ad is displayed ina web browser window of a user device. An ad also can be configured toinclude embedded code, such as a hyperlink, that allows a user to“click-through” the ad. Clicking-through an ad can activate a function,such as redirecting the browser to a different web page or displaying anadditional window. For example, a new window can be opened to provideinformation describing a product or service described in the ad to auser. A new window also can be opened to allow the user to initiate apurchase transaction.

The effectiveness of an ad can be measured in various ways. For example,the click-through rate (CTR) of an ad can be determined to identify howfrequently the ad is accessed when it is presented. The CTR of an ad canbe calculated by dividing the number of click-throughs associated withthe ad by the number of impressions of that ad over a given time period.The number of conversions associated with an ad also can be used as ameasure of effectiveness. A conversion can be any desired action takenin response to the ad, including a sale, a download, a contact, arequest, or an entry into a database. As with the CTR, the conversionrate can be determined by dividing the number of conversions attributedto an ad by the number of impressions of that ad over a given timeperiod.

The presentation and placement of ads on web pages has evolved alongwith other Internet content. Ads now typically occupy the periphery ofweb pages or are integrated with other content in the body of a webpage. For example, banner ads, expandable banner ads, tile ads, andskyscraper ads each can be mixed with additional content comprising aweb page. Ads also can be displayed in separate browser windows, whichare referred to as pop-up or pop-under windows. Further, ads can beintegrated with toolbars or presented as temporary objects that areconfigured to appear and disappear without user intervention.

As with conventional print and broadcast media, a web site can be usedto present (or “host”) ads associated with one or more third-partyadvertisers. Further, the hosting web site can receive compensation fromthe third-party running the ad, including payment and reciprocaladvertising. Additionally, a web site can demand higher payment for thepresentation of an ad as the amount of traffic on that web siteincreases. As with conventional media, the increased traffic can providegreater exposure for the ad, which in turn can generate a higher numberof conversions or click-throughs even if the CTR or conversion rate islower than that of a less popular site.

Search engines were among the first content providers to successfullysell Internet space to third-party advertisers. Further, some searchengine providers created advertising systems that are able to matchadvertisements with available advertising space, both on the searchengine's web pages and web pages associated with other contentproviders. A search engine advertising system also can be configured toconduct an automated, real-time auction to select one or more ads fordisplay in response to input received from a user. Additionally, asearch engine advertising system can be further configured toautomatically insert one or more ads into content comprising aparticular web page. For example, an ad can be inserted into apredetermined area of a static or dynamically-generated web page.Further, an advertising system can select and order ads that are to beinserted into a web page in accordance with a wide range of criteriaassociated with the ads. The criteria used to select and order ads caninclude one or more bids submitted by an advertiser, a CTR, a conversionrate, a geographical preference, and a language preference.

Additionally, an advertising system can be configured to permit multipleadvertisers to compete for the same advertising space. For example, asearch engine can display one or more ads in response to a keyword orkeyword combination received from a user. If two or more advertisersdesire to have their ads displayed in response to a specific keyword orkeyword combination, each advertiser can specify a bid corresponding totheir ad. An advertising system can be configured to select one or moreads for display in response to input from a user comprising one or morekeywords. Further, the advertising system can be configured to determinethe display order of the selected ads in accordance with one or morefactors, including the corresponding bids.

Advertisers, however, can be reluctant to allocate a portion of anadvertising budget to Internet advertising without some assuranceregarding performance. An advertiser thus can be reluctant to specify acorresponding bid that will be competitive in an auction. Therefore, avariety of ways to gauge the effectiveness of an Internet ad have beendeveloped. For example, some advertisers have launched one or more testads. A test ad can be representative of a larger advertising campaigncomprising one or more related ads. Further, the test ad can berestricted with respect to one or more parameters, including the numberof associated keywords, the allocated budget, and the length of timeover which the ad is run. Additionally, the scope of the test ad can belimited, such as to a single web site, to users in a particulargeographic region, or even with respect to the time of day and day ofthe week on which the ad is run. Data associated with the test ad can beanalyzed to determine whether an expanded campaign should be launched orwhether one or more refinements to the ad campaign should beimplemented.

The success of an ad campaign also can be projected using historicaldata. For example, the number of requests comprising a particularkeyword or keyword combination that will be received during a futureperiod can be estimated using search requests received during apreceding period of time. The CTR and/or conversion rate of an ad can besimilarly estimated using performance data associated with related adsthat have been previously run. Additionally, an initial bid amount canbe established in accordance with historical bidding levels. If datafrom a sufficient period is available, one or more campaign settingsalso can be adjusted based on a trend analysis.

Additionally, the success of an ad campaign can be modeled, such asthrough a modeling application. For example, historical data relating tosimilar ad campaigns can be manipulated based on one or more assumptionsabout the future. In order to generate projected results for an ad, oneor more parameters associated with the ad can be entered into themodeling application. The projected results produced by the model thencan be used to determine whether the ad or an associated ad campaignshould be implemented.

SUMMARY

The present inventors recognized the need to provide an improved systemand method for generating results corresponding to the performance of aproposed advertising campaign conducted in an online environment.Further, the need was recognized to use one or more actual advertisingcampaigns to assess the performance of a proposed advertising campaign.Additionally, the present inventors recognized the need to determine theresults, such as the number of impressions, clicks and conversions, aproposed advertising campaign would have generated in an advertisingnetwork if it had been implemented in accordance with one or moreparameters.

The present inventors also recognized the need to identify one or moresites on which an ad associated with a proposed advertising campaignwould have been displayed. Further, the present inventors recognized theneed to identify search results in response to which an ad associatedwith a proposed advertising campaign would have been presented.Additionally, the need was recognized to permit an advertiser toevaluate the performance of a proposed advertising campaign in responseto two or more sets of differing parameters.

The present inventors also recognized the need to automate thegeneration of results corresponding to the performance of a proposedadvertising campaign. Further, the need was recognized to generateresults corresponding to the performance of a proposed advertisingcampaign in real-time or in near real-time. Additionally, the presentinventors recognized the need to permit an ad network or an ad server toevaluate the performance of one or more advertising strategies,including newly developed advertising strategies. Accordingly, thetechniques and apparatus described here implement algorithms forgenerating results corresponding to the performance of a proposedadvertising campaign and for evaluating the results associated with aplurality of proposed and actual advertising campaigns.

In general, in one aspect, the subject matter can be implemented toinclude receiving an ad request; identifying at least one shadow ad andat least one actual ad based on the received ad request; generating anad ranking by analyzing one or more criteria associated with theidentified at least one shadow ad and the at least one actual ad; andselecting one or more of the identified at least one shadow ad and theat least one actual ad based on the ad ranking.

The subject matter also can be implemented such that the at least oneshadow ad is associated with one or more campaign parameterscorresponding to a shadow ad campaign. Further, the subject matter canbe implemented such that identifying at least one shadow ad and at leastone actual ad further comprises comparing the one or more campaignparameters associated with the at least one shadow ad with one or morecriteria included in the ad request to determine a match. Additionally,the subject matter can be implemented such that the one or more criteriaassociated with the identified at least one shadow ad and the at leastone actual ad include a performance factor.

The subject matter also can be implemented to include updating an adcampaign database in accordance with the one or more selected shadowads. Further, the subject matter can be implemented to include recordinga charge corresponding to the one or more selected shadow ads.Additionally, the subject matter can be implemented such that the adrequest is received from an online advertising system.

In general, in another aspect, the subject matter can be implemented asa computer program product, encoded on a computer-readable medium,operable to cause data processing apparatus to perform operationscomprising receiving an ad request; identifying at least one shadow adand at least one actual ad based on the received ad request; generatingan ad ranking by analyzing one or more criteria associated with theidentified at least one shadow ad and the at least one actual ad; andselecting one or more of the identified at least one shadow ad and theat least one actual ad based on the ad ranking.

The subject matter also can be implemented such that the at least oneshadow ad is associated with one or more campaign parameterscorresponding to a shadow ad campaign. Further, the subject matter canbe implemented such that identifying at least one shadow ad and at leastone actual ad further comprises comparing the one or more campaignparameters associated with the at least one shadow ad with one or morecriteria included in the ad request to determine a match. Additionally,the subject matter can be implemented such that the one or more criteriaassociated with the identified at least one shadow ad and the at leastone actual ad include a performance factor.

The subject matter also can be implemented to be further operable tocause data processing apparatus to perform operations comprisingupdating an ad campaign database in accordance with the one or moreselected shadow ads. Further, the subject matter can be implemented tobe further operable to cause data processing apparatus to performoperations comprising recording a charge corresponding to the one ormore selected shadow ads. Additionally, the subject matter can beimplemented such that the ad request is received from an onlineadvertising system.

In general, in another aspect, the subject matter can be implemented asa system comprising an ad database storing at least one shadow ad and atleast one actual ad and a shadow ad server configured to performoperations comprising receiving an ad request; identifying in the addatabase at least one shadow ad and at least one actual ad based on thereceived ad request; generating an ad ranking by analyzing one or morecriteria associated with the identified at least one shadow ad and theat least one actual ad; and selecting one or more of the identified atleast one shadow ad and the at least one actual ad based on the adranking.

The subject matter also can be implemented such that the ad databasefurther stores one or more campaign parameters associated with the atleast one shadow ad. Further, the subject matter can be implemented suchthat the operations comprising identifying at least one shadow ad and atleast one actual ad further comprise comparing the one or more campaignparameters associated with the at least one shadow ad with one or morecriteria included in the ad request to determine a match. Additionally,the subject matter can be implemented such that the one or more criteriaassociated with the identified at least one shadow ad and the at leastone actual ad include a performance factor.

The subject matter also can be implemented such that the shadow adserver is further configured to perform operations comprising updatingan ad campaign database in accordance with the one or more selectedshadow ads. Further, the subject matter can be implemented such that theshadow ad server is further configured to perform operations comprisingrecording a charge corresponding to the one or more selected shadow ads.Additionally, the subject matter can be implemented such that the shadowad server is further configured to perform operations comprisingreceiving the ad request from an online advertising system.

The techniques described in this document can be implemented to realizeone or more of the following advantages. For example, the techniques canbe implemented to identify the actual results a proposed advertisingcampaign would have generated if it had been active during a particulartime period. The results corresponding to the proposed advertisingcampaign also can be compared with one or more actual advertisingcampaigns that were active for the same time period. Further, thetechniques can be implemented to assess the effectiveness of one or moreparameters associated with a proposed advertising campaign. Thetechniques also can be implemented to permit an ad server or an adnetwork to demonstrate the effectiveness of an advertising system to anadvertiser or a potential advertiser. Additionally, the techniques canbe implemented to permit an advertiser to identify search results withwhich an ad would have been displayed or one or more web sites on whichan ad would have appeared.

Further, the techniques can be implemented to permit an advertiser togenerate results corresponding to a full advertising campaign instead ofa sample or representative campaign. The techniques also can beimplemented such that results associated with a proposed advertisingcampaign can be automatically generated in real-time or near real-time.Further, the techniques can be implemented such that results associatedwith a proposed advertising campaign can be generated based on one ormore actual advertising campaigns. Additionally, the techniques can beimplemented to reduce one or more aspects of the human cost associatedwith assessing the effectiveness of a proposed advertising campaign.

These general and specific techniques can be implemented using anapparatus, a method, a system, or any combination of an apparatus,methods, and systems. The details of one or more implementations are setforth in the accompanying drawings and the description below. Furtherfeatures, aspects, and advantages will become apparent from thedescription, the drawings, and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts a user interface presenting information associated withan advertising campaign.

FIG. 2 depicts a user interface presenting information associated withan ad group.

FIG. 3 presents a method of evaluating the effectiveness of one or moreshadow ad campaigns.

FIG. 4 presents a method of modifying a shadow ad campaign in responseto performance in an advertising system.

FIG. 5 depicts an advertising system for evaluating a shadow adcampaign.

FIG. 6 shows a flow chart for evaluating a shadow ad.

Like reference symbols indicate like elements throughout thespecification and drawings.

DETAILED DESCRIPTION

FIG. 1 presents a user interface 100 configured to display informationassociated with an advertising campaign comprising one or more adgroups. The advertising campaign can be an actual campaign in which oneor more advertisements are presented to public users. For example, anadvertisement associated with the actual ad campaign can be displayed inconjunction with search results presented in response to a searchrequest. An advertisement also can be displayed along with other contenton a third-party web page. Alternatively, the advertising campaign canbe a shadow campaign that can be used to test the performance of one ormore parameters associated with the ad campaign. The one or moreadvertisements comprising a shadow campaign do not compete forpresentation space and position with advertisements of one or moreactual campaigns. The one or more advertisements of a shadow campaignalso are not displayed (or “served”) to public users.

The user interface 100 includes a campaign name 102 that identifies thead campaign and a campaign status identifier 104 that indicates thecurrent status of that ad campaign. For example, the campaign name 102is Campaign #1 and the campaign status identifier 104 indicates thatCampaign #1 is currently Active. Therefore, an ad corresponding to an adgroup included in Campaign #1 can be selected and displayed, and a costto the advertiser sponsoring Campaign #1 can be incurred.

In an implementation, the status of an ad campaign can be active if thecampaign is presently running, paused if the campaign has beentemporarily halted, and deleted if the campaign has been permanentlydisabled. The campaign status identifier 104 also can be configured toinclude one or more selectable controls to permit a user, such as anadvertiser, to change the campaign status. For example, a pause campaignlink 106 can be included to permit a user to pause the campaign and adelete campaign link 108 can be included to permit a user to delete thecampaign. If the campaign is paused, the pause campaign link 106 can behidden or deactivated, and an activate campaign link can be displayed topermit a user to activate the paused campaign. The user interface 100also can be configured to include a campaign budget identifier 110 thatindicates the campaign budget for a given period, such as a day, a week,or a month. Further, an edit campaign settings control 112 can beincluded to permit a user to change one or more parameters associatedwith the ad campaign.

One or more of the ad groups comprising the ad campaign can be selectedfor display in the user interface 100 through the ad group menu 118.Information corresponding to each of the selected ad groups can bepresented in the ad group summary 120. Further, a period for whichinformation corresponding to the selected ad groups is to be displayedalso can be specified. A predetermined period can be selected from theperiod menu 122, such as a day, a week, a month, or all. Additionally, acustom time period can be specified through the date entry form 124.Once the desired time period for a report has been specified, the usercan select the go button 126 to populate the ad group summary 120 withinformation for that period.

The ad group summary 120 also can include control buttons to permit theuser to perform actions associated with one or more of the ad groupscomprising the ad campaign. An edit bids button 128 can be selected tomodify the bid value associated with one or more ad groups. If a userdetermines that ads associated with an ad group are not being displayedas frequently or as prominently as desired, the user can increase thevalue of the bid associated with that ad group. Alternatively, if a userdetermines that the cost attributable to an ad group are too high, theuser can decrease the value of the bid associated with that ad group. Arename button 130 also can be included to permit a user to rename anexisting ad group.

As discussed above with respect to an ad campaign, an individual adgroup within an ad campaign also can be activated, paused, or deleted.Therefore, the ad group summary 120 can include a plurality of statusbuttons to permit a user to change the status of an ad group. Forexample, a pause button 132, a resume button 134, and a delete button136 can be included in the ad group summary to permit a user to specifya status for a newly created ad group or modify the status of anexisting ad group.

The ad group summary 120 includes an ad group name 140 columnidentifying each ad group listed in the report. Further, the informationassociated with an ad group is presented as a single row of the ad groupsummary 120. For example, information associated with the ad grouparthritis is presented in a first row 160 and information associatedwith the ad group migraine is presented in a second row 165.Additionally, the bottom row 170 can be used to present a column totalor column average for all of the ad groups included in the ad groupsummary 120.

A status column 142 can be included in the ad group summary 120 toindicate the status of each ad group, such as active, paused, ordeleted. In an implementation, information associated with a deleted adgroup can remain accessible to the advertiser. A default bid column 144indicates the maximum cost-per-click (“CPC”) associated with an adgroup. The maximum CPC represents the highest amount an advertiser iswilling to pay if an advertisement associated with that ad groupgenerates a click-through. Although a bid associated with an impressionfor an ad group cannot exceed the default bid, the bid can be lower. Inan implementation, the default bid can be expressed in different terms,such as the cost-per-thousand impressions (“CPM”) instead of CPC.

The ad group summary 120 also can be configured to present informationrelating to the performance of an ad group. A clicks column 146indicates the number of clicks that have been generated by an ad groupfor the time period of the ad group summary 120. Further, an impressionscolumn 148 provides the number of ad impressions for an ad group overthe time period of the ad group summary 120. Additionally, the number ofclicks can be divided by the number of impressions to determine the CTR,which can be presented in the CTR column 150.

As the bid for an ad group can vary with each impression, an averagecost for the ad group over the time period of the ad group summary 120can be displayed in the average CPC column 152. A cost column 154 alsocan be provided to indicate the total cost of the ad group over theperiod of the ad group summary 120. Additionally, an average positioncolumn 156 can be included to identify the average display position ofthe impressions associated with the ad group. By specifying two or moredifferent ad group summary periods, a user can analyze the performanceof an ad group over time. Further, a user also can compare theperformance of two or more ad groups over identical time periods. Ifdesired, one or more ad group parameters can be modified to influencefuture ad group performance.

The user interface 100 also can include one or more controls throughwhich negative limitations can be specified for an ad campaign. Anegative keyword control 114 can be included to permit a user to specifyone or more negative keywords for an ad campaign. Once specified, noneof the advertisements included in the ad campaign will be selected inresponse to criteria that contains a negative keyword. An excluded sitecontrol 116 also can be included to permit a user to specify one or moresites on which an advertisement of the ad campaign shall not bepresented. An advertiser can thus elect not to permit the advertisementsof an ad campaign to be displayed on a site that conflicts with theadvertiser's goals or that could harm the advertiser's image, such asthe site of a competitor or a site containing objectionable material.For example, an advertiser can specify that an ad campaign directed to afamily restaurant is not to appear on a particular site featuringadult-oriented content. In an implementation, one or more of a customerlanguage, an advertising location or geographic region, a currency, anda daily budget also can be defined for an ad campaign.

FIG. 2 presents a user interface 200 configured to display informationassociated with an ad group of an advertising campaign, such as theadvertising campaign described with respect to FIG. 1. An ad group iscomprised of one or more advertisements and one or more associatedparameters. The parameters can be used by an advertising system toselect and present one or more advertisements to public users. In animplementation, if an ad campaign is a shadow campaign, the one or moreadvertisements associated with an ad group can be notional orplaceholder ads.

The user interface 200 includes an ad group identifier 205 thatindicates the name of the ad group, such as Arthritis. The userinterface 200 also can include an ad group status identifier 210 thatindicates the current status of the ad group, such as active, paused, ordeleted. The ad group status identifier 210 also can include one or morestatus controls to permit a user to change the status. For example, ifthe ad group status is active, status controls can be provided to changethe ad group status to paused or deleted.

The user interface 200 also can include a plurality of tabs that provideinformation relating to the ad group and allow a user to modify one ormore parameters associated with the ad group. For example, the userinterface 200 can include a keyword tab 215 that describes theperformance of one or more keywords corresponding to the ad group, asummary tab 220 that describes ad group performance on one or more adnetworks, and an ad variations tab 225 that describes the one or moreads comprising the ad group.

The information presented in a tab, such as the keyword tab 215, can berestricted to a particular date or date range. Thus, the user interface200 includes a date range menu 230 that provides predetermine options,such as a day, a week, a month, or all. Further, the user interface 200also includes a date entry form 235 through which a user can specify acustom period.

The user interface 200 also can include one or more controls to permit auser to modify a keyword associated with the ad group. A checkboxcorresponding to a particular keyword, such as the checkbox 237corresponding to the keyword rheumatoid, can be selected to modifiedthat keyword. For example, the user interface 200 can include an editkeyword settings button 240. In response to a user selecting the editkeyword settings button 240, the user interface 200 can display one ormore settings associated with the one or more selected keywords suchthat those settings can be edited by a user. The user interface 200 alsocan include a delete button 245 that can be used to delete the one ormore selected keywords.

The keyword tab 215 can be configured to present the informationassociated with the ad group keywords in a table or spreadsheet format.Thus, a first row 250 can identify the category of information presentedin each column. Further, the information associated with a particularkeyword is presented as a single row in the keywords tab 215. Forexample, information associated with the keyword rheumatoid is presentedin a first row 260 and information associated with the keywordosteoarthritis is presented in a second row 270. Additionally, thebottom row 280 can be used to present a column total or column averagefor all of the keywords included in the keyword tab 215.

FIG. 3 presents a method of evaluating the effectiveness of one or moreshadow ad campaigns. A shadow ad campaign includes one or more shadowads. In an implementation, a shadow ad can comprise a proposed ad thathas not been previously used, an actual ad that has been used in anactual ad campaign, or a notional ad, such as a placeholder. Each shadowad campaign also can include one or more campaign parameters that can beused to determine selection and placement of the shadow ads in responseto an ad request, including a default bid, one or more negativekeywords, and one or more excluded sites.

An ad request is received by the advertising system (305). The adrequest can be an actual request submitted by a public user or a webserver hosting one or more actual web pages. For example, a public usercan submit an actual request by entering search criteria comprising oneor more keywords into a search engine. Similarly, a web server canreceive a request for a web page and can generate an actual ad requestin the process of serving that page. Alternatively, an ad request can bea simulated request that is generated by or provided to the advertisingsystem. For example, one or more ad requests can be simulated based onan anticipated response to a future event. In an implementation, actualand simulated requests also can be filtered in accordance with one ormore predetermined criteria to limit the ad requests processed by theadvertising system.

One or more shadow ad campaigns can be analyzed to identify ads thatcorrespond to the received ad request (310). The shadow ad campaigns canbe analyzed using criteria included in the ad request and one or more adcampaign parameters. For example, one or more keywords included in thead request can be compared with one or more keywords associated with ashadow ad campaign to determine whether the shadow ad campaign matchesthe ad request. Further, each of the keywords included in the ad requestcan be compared with one or more negative keywords associated with ashadow ad campaign, and a resulting match can be used to determine thatthe shadow ad campaign does not correspond to the received ad request.Additionally, the site on which the ads are to be presented can becompared with one or more excluded sites associated with a shadow adcampaign to ensure that an ad will not be selected for presentation onan excluded site. As with the shadow ad campaigns, one or more actual adcampaigns also can be analyzed to identify ads that correspond to thereceived ad request (315). Further, the one or more actual ad campaignsalso can be evaluated to ensure that an actual ad campaign is notidentified as corresponding to an ad request that is associated with anegative keyword or an excluded site.

After the shadow ad campaigns and the actual ad campaigns have beenanalyzed, the number of identified ads can be determined (320). If morethan one ad has been identified, an auction can be conducted between theidentified shadow ads and actual ads (325). The Ad Selection and AdOrdering components of the Google™ Ad Words system is an example of anadvertising system auction. During the auction, the identified ads areranked in accordance with one or more ad campaign parameters, such as adefault bid and a daily budget. Because the ad campaigns and associatedads are used only to evaluate one or more shadow ad campaigns, ads arenot presented to public users. Therefore, the shadow ads and the actualads can be treated equally in the auction. In an implementation, one ormore additional factors also can be considered in conducting theauction. For example, the CTR and/or conversion rate of an ad can beused as an auction factor. Further, an auction factor can be weighted toincrease or decrease the impact that factor has on the ranking. An adthat has a higher CTR can thus be ranked above an ad that has a lowerCTR, even if the ad with the lower CTR has an equal or greater defaultbid.

One or more of the identified ads further can be selected forpresentation (330). The identified ads can be selected based on theranking determined during the auction. For example, if the identifiedads are to be included in a web page that has space to display four ads,the four highest ranked ads can be selected for presentation. Further,the ranking established during the auction can be used to determine adisplay order. Therefore, the highest ranked ad can be assigned to themost prominent display position. In an implementation, the display ordercan be used to determine a user action associated with a shadow ad, suchas a click-through or a conversion.

The advertiser sponsoring a shadow ad also can be charged based on adselection (335). For example, if an advertiser is charged on a CPMbasis, the shadow ad campaign corresponding to a selected ad can bedebited by the bid amount established during the auction. Alternatively,if an advertiser is charged on a CPC basis, the shadow ad campaigncorresponding to a selected ad can be debited by the bid amount if theadvertising system determines that the ad would have generated aclick-through. By debiting a shadow ad campaign for each impression orclick-through generated, the actual cost of a shadow campaign can beaccurately assessed. Because only the effectiveness of one or moreshadow ad campaigns is being evaluated, information associated withparticipating actual ad campaigns need not be tracked.

Similarly, an ad campaign database can be updated to reflect the currentinformation associated with the shadow ad campaign and ad groupcorresponding to a shadow ad (340). For example, information describingthe number of impressions, the number of clicks, the CTR, the averageCPC, the total cost, and the average position for an ad group can beupdated. Within an ad group, the same information can be updated for akeyword or a keyword combination that generated the impression. Byupdating the information associated with a shadow ad campaign, anadvertiser can accurately assess the performance of the entire shadow adcampaign as well as one or more associated parameters.

If only one ad has been identified, there is no need to conduct anauction. As only the effectiveness of one or more shadow ad campaigns isbeing evaluated, the advertising system can determine whether theidentified ad is a shadow ad (345). If the identified ad is a shadow ad,the advertiser can be charged the minimum bid for presenting the shadowad (350). As described above, the ad campaign database also can beupdated to reflect the current information associated with the shadow adcampaign and ad group corresponding to the shadow ad (355).

In an implementation, ad requests received by an online advertisingsystem that is serving actual ads to public users can be provided inreal-time or near real-time to a shadow advertising system configured toevaluate one or more shadow ad campaigns. Further, user actions detectedby the online advertising system, such as click-throughs andconversions, can be utilized to generate projected user actions for theshadow advertising system.

FIG. 4 presents a method of modifying a shadow ad campaign in responseto its performance in an advertising system. A shadow ad campaign can beexecuted in an advertising system (405). The shadow ad campaign cancomprise one or more shadow ads and one or more campaign parameters,including keywords, bids, budgets, and excluded sites. A shadow adcampaign can be a newly devised campaign that has not been previouslyimplemented. Alternatively, a shadow ad campaign can be an existingcampaign that has been previously implemented, such as an inactivecampaign that an advertiser would like to reintroduce.

An advertising system can be configured to include the shadow adcampaign in a plurality of auctions conducted using one or more othershadow ad campaigns and/or actual ad campaigns. Further, the advertisingsystem can conduct the plurality of auctions as shadow auctions thatutilize actual and/or projected ad requests to evaluate the performanceof the shadow ad campaign. In another implementation, the advertisingsystem can include the shadow ad campaign in a plurality of actualauctions conducted on an advertising system that serves actual ads inresponse to received ad requests. If a shadow ad is selected in anactual auction, the shadow ad can be filtered out prior to thepresentation of one or more ads to a public user.

The performance of a shadow ad campaign can be analyzed to assess itseffectiveness (410). A number of performance factors can be considered,including the number of impressions generated, the number of clicksreceived, the total cost over the analysis period, the average cost perimpression or click, and the average display position of an associatedad. Further, the performance of the shadow ad campaign can be analyzedafter a predetermined period of time, such as after an evaluation periodhas elapsed. The performance of the shadow ad campaign also can beanalyzed as desired by an evaluator, such as the advertiser or anadvertising system administrator. Additionally, the analysis can beperformed manually by the evaluator or automatically, such as by theadvertising system.

Once the effectiveness of the shadow ad campaign has been assessed, oneor more performance goals can be evaluated (415). For example, anadvertiser can desire a minimum CTR or conversion rate for a particulardefault bid. If the one or more performance goals are met, the campaignparameters associated with the shadow ad campaign can be saved (420).The shadow ad campaign then can be implemented as an actual ad campaignat the direction of the advertiser.

If one or more performance goals associated with the shadow ad campaignare not met, one or more campaign parameters can be revised (425). Anevaluator can manually revise a campaign parameter, such as byspecifying a new campaign parameter through a user interface.Alternatively, a campaign parameter can be revised automatically inresponse to a predetermined performance goal. Thus, the advertisingsystem can be configured to attempt to optimize one or more aspects of ashadow ad campaign over time by revising one or more campaignparameters. For example, the advertising system can be configured toidentify the lowest default bid that will result in an ad being rankedin the second display position.

Further, one or more feedback loops can be implemented to revise acampaign parameter or a combination of campaign parameters. For example,the number of impressions generated by an ad campaign can be logicallycoupled to the default bid. If the number of impressions does notsatisfy a predetermined minimum value, the default bid can beautomatically increased. Similarly, a maximum daily budget and the CTRcan be logically coupled to a parameter specifying the time of dayduring which the shadow ad campaign is to be run. The resulting feedbackloop can be utilized to select a daily time period over which the shadowad campaign generates a sufficiently high CTR for the predeterminedbudget. Additionally, a feedback loop used to revise a campaignparameter can be stabilized, such as through the use of discrete Bodeplots, Nyquist criteria, or nonlinear damping functions.

FIG. 5 depicts an advertising system 500 for evaluating a shadow adcampaign. The advertising system 500 includes an ad server 505configured to serve one or more ads to an ad consumer 510. The ad server505 can receive an ad request from the ad consumer 510 over a networkconnection 515. The ad request can comprise a user defined request, suchas one or more search criteria submitted by a user, or an automatedrequest, such as a request to fill advertising space on a web page thatis being loaded for display to a user.

Upon receiving an ad request from the ad consumer 510, the ad server 505can access an operational ad database 520 that is configured to storeone or more actual ads intended for presentation to users. Further, oneor more actual ads can be associated to form an actual ad campaign. Thead server 505 can be configured to search the actual ads stored in theoperational ad database 520 and to identify actual ads that correspondto the ad request. For example, the ad server 505 can identify actualads associated with one or more keywords included in the ad request. Thead server 505 also can be configured to rank the actual ads that havebeen identified according to one or more criteria, such as a default bidor a conversion rate. After the ranking has been determined, the adserver 505 can provide one or more actual ads to the ad consumer 510 tosatisfy the ad request. For example, if the ad request specified a needfor three ads, the three highest ranked actual ads can be transmitted tothe ad consumer 510.

The advertising system 500 also can include a shadow ad server 525 thatcan be configured to conduct shadow auctions and to maintain one or moreshadow ad campaigns. Further, the shadow ad server 525 can be configuredto communicate with the ad server 505 over a communication channel 522,such as a local area network. The shadow ad server 525 can thus accessactual ads stored in the operational ad database 520 and receive adrequests submitted to the ad server 505 by the ad consumer 510.

The shadow ad server 525 can conduct a shadow auction using at least oneshadow ad campaign stored in a shadow ad database 530 and one or moreactual ad campaigns stored in the operational ad database 520. Further,a shadow auction can be conducted based on an ad request submitted bythe ad consumer 510. Alternatively, a shadow auction can be conductedusing a simulated ad request. For example, one or more ad requests canbe simulated for use by the shadow ad server 525 to produce anadvertising environment based on one or more assumptions about thefuture. In an implementation, a plurality of shadow auctions can beconducted to evaluate the effectiveness of a shadow ad campaign over aperiod of time using ad requests received from the ad consumer 510and/or simulated ad requests. In another implementation, a shadowauction can be conducted by the shadow ad server 525 in parallel with anactual auction conducted by the ad server 505 to obtain shadow auctionresults in real-time or near real-time.

One or more items of information associated with a shadow ad campaignstored in the shadow ad database 530 can be updated to reflect theresults of a shadow auction. Further, information associated with one ormore shadow ad campaigns can be presented to a user, such as anadvertiser, through a user interface 535. The user interface 535 can beprovided on the shadow ad server 525. Alternatively, the user interface535 can comprise a separate computing platform that is coupled to theshadow ad server 525 through a communication channel 532, such as theInternet. Further, one or more items of information associated with ashadow ad campaign stored in the shadow ad database 530, includingcampaign parameters, can be modified through the user interface 535.

In an implementation, the shadow ad server 525 and the ad server 505 cancomprise a single computing platform. Further, the shadow ad database530 and the operational ad database 520 can be hosted on a singlecomputing platform. Additionally, the shadow ad database 530 and theoperational ad database 520 can be combined into a single databasestructure, such that an actual ad campaign is differentiated from ashadow ad campaign through one or more fields or other such logicalidentifiers.

FIG. 6 shows a flow chart for evaluating a shadow ad. Initially, an adrequest can be received (605). At least one shadow ad and at least oneactual ad can be identified based on the received ad request (610). Anad ranking can be generated by analyzing one or more criteria associatedwith the identified at least one shadow ad and the at least one actualad (615). Once the ad ranking has been generated, one or more of theidentified at least one shadow ad and the at least one actual ad can beselected based on the ad ranking (620).

A number of implementations have been disclosed herein. Nevertheless, itwill be understood that various modifications may be made withoutdeparting from the spirit and scope of the claims. Accordingly, otherimplementations are within the scope of the following claims.

1-20. (canceled)
 21. A computer-implemented method comprising:identifying a shadow content item, the shadow content item not beingeligible for an auction for selecting content items to be served inresponse to a received request for content; receiving a request forcontent; evaluating the inventory of content items to identify candidatecontent items in the inventory to serve responsive to the request,including ranking the candidate content items and selecting ahighest-ranking candidate content item; determining a relative rankingthat ranks the highest-ranking candidate content item against the shadowcontent item based at least in part on the request; determining ameasure of performance of the shadow content item based at least in parton the relative ranking between the highest-ranking candidate contentitem and the shadow content item; determining that the measure ofperformance of the shadow content item did not meet a performance goal;and in response to determining that the measure of performance of theshadow content item did not meet a performance goal, automaticallyadjusting a parameter associated with the shadow content item.
 22. Themethod of claim 1, further comprising: executing a shadow campaignaccording to one or more parameters associated with the shadow campaign,including determining one or more relative rankings that rank the shadowcontent item against one or more candidate content items.
 23. The methodof claim 2, wherein the performance goal for the shadow content itemcomprises a minimum click-through rate or a minimum conversion rate fora default bid over the shadow campaign.
 24. The method of claim 2,further comprising: executing the shadow campaign according to theadjusted parameter of the shadow content item, including determining oneor more relative rankings that rank the shadow content item against oneor more candidate content items according to the adjusted parameter ofthe shadow content item; determining a second measure of performance ofthe shadow content item based at least in part on the one or morerelative rankings; determining that the second measure of performance ofthe shadow content item meets the performance goal; and saving theadjusted parameter in association with the shadow content item.
 25. Themethod of claim 4, further comprising: receiving a request to executethe shadow campaign as an actual campaign; and executing the shadowcampaign as an actual campaign using the adjusted parameter of theshadow content item, including converting the shadow content item intocandidate content item that is eligible to be served in response torequests for content.
 26. The method of claim 1, wherein determining therelative ranking comprises conducting a shadow auction based on a firstbid associated with the shadow content item and a second bid associatedwith the highest-ranking candidate content item.
 27. The method of claim6, wherein the shadow auction is conducted in response to the requestfor content on a content page.
 28. The method of claim 1, furthercomprising: determining that the shadow content item would be servedresponsive to the request for content; and in response to determiningthat the shadow content item would be served responsive to the requestfor content, recording a charge in shadow campaign data associated withthe shadow content item.
 29. The method of claim 1, whereinautomatically adjusting the parameter associated with the shadow contentitem comprises increasing a bid amount associated with the shadowcontent item.
 30. A system comprising: one or more computers and one ormore storage devices storing instructions that are operable, whenexecuted by the one or more computers, to cause the one or morecomputers to perform operations comprising: identifying a shadow contentitem, the shadow content item not being eligible for an auction forselecting content items to be served in response to a received requestfor content; receiving a request for content; evaluating the inventoryof content items to identify candidate content items in the inventory toserve responsive to the request, including ranking the candidate contentitems and selecting a highest-ranking candidate content item;determining a relative ranking that ranks the highest-ranking candidatecontent item against the shadow content item based at least in part onthe request; determining a measure of performance of the shadow contentitem based at least in part on the relative ranking between thehighest-ranking candidate content item and the shadow content item;determining that the measure of performance of the shadow content itemdid not meet a performance goal; and in response to determining that themeasure of performance of the shadow content item did not meet aperformance goal, automatically adjusting a parameter associated withthe shadow content item.
 31. The method of claim 30, wherein theoperations further comprise executing a shadow campaign according to oneor more parameters associated with the shadow campaign, includingdetermining one or more relative rankings that rank the shadow contentitem against one or more candidate content items.
 32. The system ofclaim 31, wherein the performance goal for the shadow content itemcomprises a minimum click-through rate or a minimum conversion rate fora default bid over the shadow campaign.
 33. The system of claim 31,wherein the operations further comprise: executing the shadow campaignaccording to the adjusted parameter of the shadow content item,including determining one or more relative rankings that rank the shadowcontent item against one or more candidate content items according tothe adjusted parameter of the shadow content item; determining a secondmeasure of performance of the shadow content item based at least in parton the one or more relative rankings; determining that the secondmeasure of performance of the shadow content item meets the performancegoal; and saving the adjusted parameter in association with the shadowcontent item.
 34. The system of claim 33, wherein the operations furthercomprise: receiving a request to execute the shadow campaign as anactual campaign; and executing the shadow campaign as an actual campaignusing the adjusted parameter of the shadow content item, includingconverting the shadow content item into candidate content item that iseligible to be served in response to requests for content.
 35. Thesystem of claim 30, wherein determining the relative ranking comprisesconducting a shadow auction based on a first bid associated with theshadow content item and a second bid associated with the highest-rankingcandidate content item.
 36. The system of claim 35, wherein the shadowauction is conducted in response to the request for content on a contentpage.
 37. The system of claim 30, wherein the operations furthercomprise: determining that the shadow content item would be servedresponsive to the request for content; and in response to determiningthat the shadow content item would be served responsive to the requestfor content, recording a charge in shadow campaign data associated withthe shadow content item.
 38. The system of claim 30, whereinautomatically adjusting the parameter associated with the shadow contentitem comprises increasing a bid amount associated with the shadowcontent item.
 39. A computer program product, encoded on one or morenon-transitory computer storage media, comprising instructions that whenexecuted by one or more computers cause the one or more computers toperform operations comprising: identifying a shadow content item, theshadow content item not being eligible for an auction for selectingcontent items to be served in response to a received request forcontent; receiving a request for content; evaluating the inventory ofcontent items to identify candidate content items in the inventory toserve responsive to the request, including ranking the candidate contentitems and selecting a highest-ranking candidate content item;determining a relative ranking that ranks the highest-ranking candidatecontent item against the shadow content item based at least in part onthe request; determining a measure of performance of the shadow contentitem based at least in part on the relative ranking between thehighest-ranking candidate content item and the shadow content item;determining that the measure of performance of the shadow content itemdid not meet a performance goal; and in response to determining that themeasure of performance of the shadow content item did not meet aperformance goal, automatically adjusting a parameter associated withthe shadow content item.
 40. The computer program product of claim 39,wherein the operations further comprise: executing a shadow campaignaccording to one or more parameters associated with the shadow campaign,including determining one or more relative rankings that rank the shadowcontent item against one or more candidate content items.